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Abstract 

The  current  methodologies  used  in  risk  assessment  are  heavily  subjective  and  inaccurate  in 
various  life  cycle  phases  of  complex  engineered  systems.  The  increase  in  complexity  has 
caused  a  paradigm  shift  from  root  cause  analysis  to  the  search  of  a  set  of  concurrent  causes 
for  each  event  and  the  relevant  complexity  content  of  the  system.  Many  of  the  system’s  life 
cycle  risks  are  currently  assessed  subjectively  by  imprecise  methodologies  such  as  color- 
coded  risk  matrix,  and  subsequently  they  suffer  from  unforeseen  failures  as  well  as  cost  and 
schedule  overruns.  This  research  project  proposes  a  novel  approach  to  major  improvement 
of  risk  assessment  by  creating  a  set  of  appropriate  complexity  measures  (informed  by 
historical  case  studies)  as  pre-indicators  of  emergence  of  risks  at  different  stages  of  a 
systems  development  process,  and  also  a  framework  that  enables  the  decision-makers  on 
assessing  the  actual  risk  level  at  each  phase  of  the  development  based  on  requirements, 
design  decisions,  and  alternatives.  The  goal  of  this  research  is  to  capture  the  complexity  of 
the  system  with  some  innovative  metrics,  thus  allowing  for  better  decision-making  in 
architecture  and  design  selections. 

Introduction 

Engineered  systems  have  become  progressively  more  complex  and  interconnected 
to  other  various  infrastructure  systems  over  the  past  few  decades,  and  they  continue  to 
become  more  complex.  Examples  of  this  can  be  seen  in  various  fields  of  engineered 
systems,  spanning  from  satellites,  aircrafts,  and  missiles  to  ground  transportation  systems 
and  sophisticated  interconnected  power  and  communication  grids.  In  one  perspective,  more 
complexity  provides  more  sophisticated  multi-functionality  to  the  engineered  system  at  hand, 
while  in  a  competing  perspective,  concurrently  can  make  the  system  more  vulnerable  and 
fragile  and  prone  to  failures  and  emergent  behavior.  The  relationship  between  excessive 
complexity  in  design  and  operation  of  complex  engineered  systems  to  the  risk,  emergence, 
and  increased  manifestation  of  failures  has  been  acknowledged  by  many  experts  and 
academics  in  various  engineering  design  communities.  However,  there  is  a  lack  of 
comprehensive  research  that  enables  the  discovery  of  the  relationship  between  the  level  of 
complexity  of  a  design  to  increased  risks  and  failure  of  that  system.  This  research  is  an 
initial  study  in  understanding,  modeling,  and  suggesting  relevant  complexity  measures  in 
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engineering  design  that  can  be  used  and  linked  quantitatively  to  the  risk  assessment  of  an 
engineered  system. 
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Figure  1.  Traditional  Risk  Reporting  Matrix 

(DoD,  2006) 

Risk  can  be  defined  as  “a  measure  of  future  uncertainties  in  achieving  program 
performance  goals  and  objectives  within  defined  cost,  schedule  and  performance 
constraints”  (DoD,  2006).  In  complex  engineered  systems  as  well  as  acquisition  programs, 
often  various  types  of  risks  exist  that  manifest  themselves  at  different  times  throughout  the 
development  process.  These  risks  can  be  technical,  programmatic,  or  strategic  in  nature 
and  can  result  in  substantial  cost  overruns,  delays,  performance  issues,  reduced 
adaptability  to  changing  requirements,  or  even  total  cancellation  of  a  project.  The  major 
challenges  of  assessing  risk  using  the  traditional  risk  reporting  matrices  (Figure  1)  for 
complex  systems  acquisition  is  that  neither  the  likelihood  nor  the  true  consequence  of  a  risk 
can  be  objectively  established.  Substantial  uncertainty  around  the  interactions  among 
different  components  of  a  system  as  well  as  uncertainties  across  a  multiplicity  of  interfaces. 
Also,  often  the  symptoms  and  events  after  a  failure  or  a  problem  manifest  itself  can  be  seen 
and  are  visible  (Figure  2);  however,  the  behavior  and  structure  of  the  engineered  system 
and  the  architecture  and  level  of  complexity  of  the  engineered  system  that  gives  rise  to  such 
unforeseen  events  are  often  unknown.  By  making  the  complexity  content  and  the 
architectural  pattern  of  an  engineered  system  known  and  explicit,  in  the  next  step  of 
research  we  will  be  able  to  find  the  relationship  between  the  underlying  structure  and 
complexity  and  the  manifestation  of  risks  and  uncertainties  in  engineered  systems. 
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Figure  2.  Problem  Statement  and  Assessment  of  Structural  Complexity  as  an 

Indicator  of  Risk  and  Failure  Emergence 

The  objective  of  this  research  project  is  to  create  a  quantitative  and  more  objective 
assessment  of  technical  risks  and  failures  in  engineered  systems.  This  research  aims  to 
explore,  formulate,  and  model  the  complex  risks  and  failure  mechanisms  to  improve  the 
current  inaccurate  subjective  assessment  of  risk  in  different  stages  of  an  engineered  system 
development  program  as  well  as  acquisition  programs. 

Literature  Review 

In  this  section  of  the  paper,  an  overview  of  the  current  literature  and  state  of  the  art 
of  the  complexity  and  complexity  measurement  of  engineered  systems  as  well  as  an 
overview  of  the  literature  on  risk  assessment  of  the  complex  engineered  systems  will  be 
discussed  briefly  to  provide  a  background  of  the  current  ongoing  research  by  the  authors. 
The  literature  review  section  begins  with  an  overview  of  complex  systems  concepts, 
followed  by  various  definitions  of  complexity  and  emergence,  several  current  existing 
measures  that  are  often  being  used  in  engineering  systems  designs.  The  section  also 
presents  a  brief  overview  of  risk  assessment  of  complex  engineered  systems. 

Risk  Management  of  Complex  Engineered  Systems 

It  is  not  possible  to  know  exactly  how  a  particular  design  will  perform  until  it  is 
built.  But  the  product  cannot  be  built  until  the  design  is  selected.  Thus,  design 
is  always  a  matter  of  decision  making  under  conditions  of  uncertainty  and 
risk.  (Hazelrigg,  1998) 

Risk  and  uncertainty  are  the  hallmarks  of  all  complex  engineered  systems.  The 
Department  of  Defense  in  the  DoD  Risk  Management  Guide  (DoD,  2006)  defines  risk  as 
follows: 


Risk  is  a  measure  of  future  uncertainties  in  achieving  program  performance 
goals  and  objectives  within  defined  cost,  schedule  and  performance 
constraints.  Risk  can  be  associated  with  all  aspects  of  a  program  (e.g.,  threat, 
technology  maturity,  supplier  capability,  design  maturation,  performance 
against  plan).  ...  Risk  addresses  the  potential  variation  in  the  planned 
approach  and  its  expected  outcome. 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-233- 


In  general,  risks  have  three  components,  which  are  the  root  cause,  a  probability  (or 
likelihood)  assessed  at  the  present  time  of  the  root  cause  occurring,  and  the  consequence 
(or  effect)  of  occurrence.  Often  a  root  cause  is  the  most  basic  reason  for  the  presence  of  a 
risk.  Accordingly,  risks  should  be  tied  to  future  root  causes  and  their  effects  (DoD,  2006). 

In  any  complex  technical  engineering  project,  risk  can  be  classified  as  either  of 
technical  or  programmatic  nature,  the  former  concerning  performance  criteria  and  the  latter 
focusing  on  cost  and  schedule.  Both  types  of  risk  are  often  modeled  as  the  product  of  the 
probability  of  an  event  and  its  severity  (Pennock  &  Haimes,  2002).  In  modeling  risk,  one  can 
also  consider  the  future  root  cause  (yet  to  happen)  of  a  certain  event  (Nilchiani  et  al.,  2013), 
which  is  where  one  is  supposed  to  act  in  order  to  eliminate  a  specific  risk.  Severity  and 
probability  are  traditionally  represented  on  the  widely  utilized,  color-coded,  risk  matrix. 

Figure  1  shows  a  color-coded  risk  matrix.  Unfortunately,  this  seemingly  quantitative  tool 
hides  subjectivity  in  the  estimation  of  event  frequency  and  severity,  and  for  those  reasons  is 
“inapt  for  today’s  complex  systems”  (Hessami,  1999).  This  not  only  means  that  most  of  the 
systems  that  we  build  today  cannot  be  built  with  the  tools  and  processes  from  last  century, 
but  also  that  we  have  started  building  in  a  domain  where  structural  patterns  matter, 
especially  for  large  projects. 

Complex  Systems 

Complexity  has  been  one  of  the  characteristics  of  many  large-scale  engineered 
systems  of  the  past  century.  Complex  engineered  systems  can  provide  sophisticated 
functionality  as  one  side  of  the  coin,  and  the  other  side  can  cause  the  system  to  be  more 
prone  to  unwanted  emergent  behaviors  and  more  fragility  to  the  engineered  system.  The 
field  of  complexity  is  rich  and  spans  over  the  past  half  century  in  various  fields  of  knowledge 
ranging  from  biological  systems  to  cyber-physical  systems.  As  it  has  been  discussed  by 
several  researchers,  a  strong  correlation  can  be  observed  between  the  complexity  of  the 
system  and  various  ranges  of  failures,  including  catastrophic  failures  (Cook,  1998;  Bar-Yam, 
2003;  Merry  &  Kassavin,  1995). 

In  1948,  Warren  Weaver,  a  pioneer  in  classifying  and  defining  complexity  in  systems, 
described  three  distinct  types  of  problems:  problems  of  simplicity,  problems  of  disorganized 
complexity,  and  problems  of  organized  complexity  (Weaver,  1948). 

According  to  Weaver  (1948),  problems  of  simplicity  are  the  problems  with  a  low 
number  of  variables  that  have  been  tackled  in  the  19th  century.  An  example  is  the  classical 
Newtonian  mechanics,  where  the  motion  of  a  body  can  be  described  with  differential 
equations  in  three  dimensions.  In  these  problems,  the  behavior  of  the  system  is  predicted  by 
integrating  equations  that  describe  the  behavior  of  its  components.  In  the  same  article, 
Weaver  discusses  that  problems  of  disorganized  complexity  are  the  ones  with  a  very  large 
number  of  variables  that  have  been  tackled  in  the  twentieth  century.  The  most  immediate 
example  is  the  motion  of  gas  particles,  or  as  an  analogy  the  motion  of  a  million  balls  rolling 
on  a  billiard  table.  The  statistical  methods  developed  are  applicable  when  particles  behave 
in  an  unorganized  way  and  their  interaction  is  limited  to  the  time  they  touch  each  other, 
which  is  very  short.  In  these  problems  it  has  been  possible  to  describe  the  behavior  of  the 
system  without  looking  at  its  components  or  the  interaction  among  them. 

Problems  of  organized  complexity  are  the  ones  that  are  to  be  tackled  in  the  21st 
century,  and  ones  that  see  many  variables  showing  the  feature  of  organization.  These 
problems  have  variables  that  are  closely  interrelated  and  influence  each  other  dynamically. 
This  high  level  of  interaction  that  gives  rise  to  organization  is  the  reason  that  these  problems 
cannot  be  solved  easily.  Weaver  described  them  as  solvable  with  the  help  of  powerful 
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calculators,  but  today’s  technology  is  not  yet  able  to  solve  the  most  complex  of  these 
problems.  These  are  the  problems  that  nowadays  we  define  as  “complex.” 

Predicting  the  behavior  of  a  system  with  many  interconnected  parts  changing  their 
behavior  according  to  the  state  of  other  components  is  a  problem  of  organized  complexity, 
and  the  system  itself  is  a  referred  to  as  a  complex  system. 

Cotsaftis  (2009)  gives  a  way  of  determining  whether  a  system  is  simple,  complicated, 
or  complex  by  looking  at  its  network  model  (i.e.,  nodes  and  edges).  The  model  defines  three 
types  of  edges:  a  free  flight  state  vertex  Vu,  a  driven  state  from  outer  source  vertex  Vie,  and 
an  interactive  state  with  other  system  components  vertex  Fi;-.  The  edges  are  channels  along 
which  there  is  a  resource  flux  pu,  pie,  or  pij.  When 

Pii»  VijMPijn  (1) 

the  ith  component  is  weakly  coupled  with  the  others,  external  and  internal.  The  dynamics  of 
the  component  can  in  this  case  be  considered  independent  from  the  other  components.  If 
the  majority  of  the  components  satisfy  inequality  (1)  the  system  is  considered  to  be  simple. 
When 


Vie  »  Pu,  inf Pij,  (2) 

the  ith  component  is  depending  on  outside  sources.  The  system  can  still  be  partitioned  in  a 
set  of  weakly  connected  subsystems  which  dynamics  is  determined  from  outside  sources.  If 
the  majority  of  the  components  satisfy  inequality  (2)  the  system  is  considered  to  be 
complicated.  When 

inf  Pij  »  Pa, Pie  (3) 

the  ith  component  is  strongly  connected  to  the  others,  and  its  dynamics  cannot  be 
determined  without  considering  the  effects  of  the  other  components.  Also,  the  manipulation 
of  the  system  cannot  be  performed  as  in  the  previous  cases,  since  the  internal  connections 
create  conditions  that  reduce  the  number  of  degrees  of  freedom.  A  system  with  a  reduced 
number  of  external  control  dimensions  that  satisfies  inequality  (3)  is  said  to  be  complex. 

This  definition  is  rather  qualitative,  since  not  all  the  nodes  in  the  system  have  the 
same  importance  (in  terms  of  connection  number  and  intensity)  and  therefore  it  makes  no 
sense  to  consider  the  majority.  For  this  reason  Cotsaftis  defines  the  index  of  complexity  as 
Cs  =  n/N,  where  n  is  the  number  of  components  that  satisfy  inequality  (3)  and  N  is  the  total 
number  of  components.  A  complicated  system  has  Cs  =  0.  Cs  =  1  corresponds  to  the  most 
complex  system  possible,  but  it  is  also  a  system  where  external  connections  are  negligible, 
and  therefore  the  system  is  isolated.  This  is  due  to  the  fact  that  a  complex  system  is 
describable  with  a  low  number  of  parameters  if  seen  from  outside,  but  has  high  connectivity 
in  its  internal  structure. 

Considering  as  an  example  a  sheepdog  and  a  herd  of  cattle,  we  realize  that  the  dog 
has  only  two  degrees  of  freedom  while  the  herd  has  2 n,  where  n  is  the  number  of  animals  in 
the  herd.  By  pushing  the  cattle  together,  the  dog  increases  their  interactions  and  decreases 
the  number  of  degrees  of  freedom  of  the  herd  to  only  two,  therefore  being  able  to  control  it. 

The  research  from  these  two  authors  has  shown  us  how  complexity  and  simplicity 
are  interrelated  concepts,  somehow  opposite,  but  that  can  also  be  found  in  the  same  system 
at  the  same  time,  depending  on  the  point  of  view.  Madni  made  a  distinction  between 
systemic  elegance,  which  “thrives  on  simplicity  through  minimalistic  thinking  and  parsimony” 
and  perceived  elegance,  which  “hides  systemic  or  organizational  complexity  from  the  user.” 

If  the  system  is  considered  to  be  complex  but  its  complexity  can  be  somehow  hidden  or 
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resolved,  thus  making  it  simpler,  then  the  design  can  be  considered  elegant  (Madni,  2012). 
Therefore,  in  order  to  achieve  a  more  elegant  design,  we  need  to  decrease  the  complexity 
of  the  system. 

Emergence 

Emergence  is  a  major  phenomenon  related  to  complex  engineered  systems. 
Emergence  at  the  macro-level  is  not  hard-coded  at  the  micro-level  (Page,  1999).  One 
example  of  emergence  in  natural  systems  is  wetness.  Water  molecules  can  be  arranged  in 
three  different  phases  (i.e.,  solid,  liquid,  and  gas),  but  only  one  of  them  expresses  a 
particular  type  of  behavior,  which  is  high  adherence  to  surfaces.  This  behavior  is  due  to  the 
intermolecular  hydrogen  bonds  that  affect  the  surface  tension  of  water  drops.  These  bonds 
are  also  active  in  the  solid  and  liquid  phase,  but  in  those  cases  they  are  either  too  strong  or 
too  weak  to  generate  wetness.  In  this  case,  the  emergence  of  a  property,  such  as  wetness, 
has  been  explained  at  a  lower  level  by  looking  at  the  molecules  that  make  up  the  liquid. 

According  to  Kauffmann  (2007),  two  different  types  of  emergence  exists  (Kauffman, 
2007).  The  reductionist  approach  sees  emergence  as  epistemological,  meaning  that  the 
knowledge  about  the  systems  is  not  yet  adequate  to  describe  the  emergent  phenomenon, 
but  it  can  improve  and  explain  it  in  the  future.  This  is  the  case  of  wetness,  where  knowledge 
about  molecules  and  intermolecular  interactions  has  explained  the  phenomenon.  On  the 
other  hand,  there  is  the  ontological  emergence  approach,  which  says  that  “not  only  do  we 
not  know  if  that  will  happen,  [but]  we  don’t  even  know  what  can  happen,”  meaning  that  there 
is  a  gap  to  fill  not  only  about  the  outcome  of  an  experiment  (or  process),  but  also  about  the 
possible  outcomes. 

Longo  presents  this  view  with  the  example  of  the  swimming  bladder  in  fishes  (Longo, 
Montevil,  &  Kauffman,  2012).  An  organ  that  gives  neutral  buoyancy  in  the  water  column  as 
its  main  function,  also  enables  the  evolution  of  some  kinds  of  worms  and  bacteria  that  will 
live  in  it.  Ontological  (or  radical)  emergence  is  given  by  the  enormous  amount  of  states  the 
system  could  evolve  into.  In  these  cases  we  not  only  are  not  able  to  predict  which  state  will 
happen,  but  we  do  not  even  know  what  the  possible  states  are. 

Gell-Mann  also  pointed  out  this  difference  using  the  concept  of  logical  depth  (Gell- 
Mann,  1995).  When  some  apparently  complex  behavior  can  be  expressed  with  simpler  laws 
that  reside  at  a  lower  level  (e.g.,  the  complicated  pattern  of  energy  levels  of  atomic  nuclei 
that  can  be  described  at  the  subatomic  level),  the  phenomenon  is  said  to  have  a  substantial 
amount  of  logical  depth. 

In  our  research,  the  emergence  that  is  going  to  be  tackled  is  considered  to  be 
epistemological  emergence,  logical  depth  according  to  Gell-Mann,  where  knowledge  about 
the  system  organizational  patterns  and  internal  structure  can  lead  to  the  explanation  of 
certain  phenomena.  Unfortunately  this  concept  is  not  so  common  in  the  systems 
engineering  and  risk  management  fields,  and  therefore  this  research  adopts  the  industry 
jargon  by  talking  about  complexity  and  complex  systems,  but  always  reminding  that  we  are 
actually  trying  to  unravel  logical  depth  from  a  systems  engineering  perspective. 

Definitions  and  Measures  of  Complexity 

There  are  various  definition  of  complexity  that  have  roots  in  various  fields  spanning 
from  mathematics  and  biology  to  engineering  design.  In  a  recent  paper,  Wade  (2014) 
suggests  that  existing  complexity  definitions  belong  to  one  of  three  types:  behavioral, 
structural,  or  constructive.  Behavioral  definitions  view  the  system  as  a  black  box  and  the 
measures  of  complexity  are  given  based  on  the  outputs  of  the  system.  Structural  definitions 
look  at  the  internal  structure  or  architecture  of  the  system.  Constructive  definitions  see 
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complexity  as  the  difficulty  in  determining  the  system  outputs  (Wade  &  Heydari,  2014).  In 
this  research  we  are  interested  in  the  modeling  behavioral  and  structural  complexity  metrics. 
A  summary  of  behavioral  complexity  definition  as  well  as  structural  complexity  and  some 
measures  are  presented  in  the  following  sections  of  the  literature  review. 

Behavioral  Complexity  Definitions  and  Metrics 

The  most  famous  behavioral  complexity  metric  is  with  no  doubt  Shannon’s  entropy 
(Shannon,  1948).  This  metric  evaluates  the  complexity  by  measuring  the  entropy  of  the 
output  message  of  the  system  (this  metric  was  initially  applied  to  information  systems). 

Gell-Mann  used  Shannon’s  entropy  to  define  information  measure  as  a  metric 
capable  of  measuring  both  the  effective  complexity,  which  is  the  amount  of  information 
necessary  to  describe  the  identified  regularities  of  an  entity,  and  the  total  information,  which 
also  takes  into  account  the  apparently  random  features  (Gell-Mann  &  Lloyd,  1996). 
Algorithmic  information  content  and  Shannon  entropy  are  used  to  build  this  metric.  The 
former  is  responsible  for  measuring  the  effective  complexity  (knowledge),  and  the  latter  the 
random  parts  (ignorance).  This  dual  approach  is  an  interesting  contribution  to  the 
measurement  of  complexity,  since  it  allows  one  to  group  similar  entities  according  to  their 
effective  complexity  and  to  measure  the  diversity  of  the  ensemble  as  entropy. 

Chaisson  (2004)  proposed  a  specific  energy-based  measure  of  complexity — more 
precisely,  energy  rate  density,  which  is  “the  amount  of  energy  available  for  work  while 
passing  through  a  system  per  unit  time  and  per  unit  mass”  (Chaisson,  2015).  This  metric 
looks  at  the  system  as  a  black  box  and  measures  the  net  energy  amount  entering  the 
system.  It  has  been  evaluated  for  multiple  entities  such  as  galaxies,  stars,  planets,  plants, 
animals,  societies,  and  technological  systems,  and  also  has  been  mapped  throughout  their 
lifetime  showing  an  increase  in  complexity  (Chaisson,  2014). 

Willcox  et  al.  (201 1 )  defined  complexity  as  “the  potential  of  a  system  to  exhibit 
unexpected  behavior  in  the  quantities  of  interest,  regardless  of  whether  or  not  that  behavior 
is  detrimental  to  achieving  system  requirements.”  She  proposed  an  entropy  and  probability 
based  metric: 


C(Q)  =  exp  (fi(X)) 


(4) 


where  X  is  the  joint  distribution  of  the  quantities  of  interest,  and  h(X)  is  the  differential 
entropy  of  X  defined  as 


(5) 


where  D.x  is  the  support  of  X. 

Structural  Complexity  Definitions  and  Metrics 

There  are  a  few  structural  complexity  measures  in  current  complex  engineering 
systems  in  recent  decades.  The  metric  presented  by  Cotsaftis  (2009)  is  an  example  of 
structural  complexity  metric,  since  it  looks  at  the  internal  structure  of  the  system  (i.e., 
components  and  interfaces). 

Another  structural  complexity  metric  was  presented  by  McCabe  for  software  systems 
(McCabe,  1976).  The  representation  of  computer  programs  using  graphs  allows  one  to 
define  the  cyclomatic  number  v(G )  as 


v(G)  =  e  —  n  +  2p 


(6) 
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where  G  is  the  graph,  e  is  the  number  of  edges,  n  is  the  number  of  nodes,  and  p  is  the 
number  of  connected  components.  This  same  metric  has  been  extended  to  measure 
architectural  design  complexity  of  a  system  (McCabe  &  Butler,  1989). 

Sinha  presented  a  structural  complexity  metric  that  uses  the  design  structure  matrix 
(DSM)  of  a  system  to  evaluate  its  complexity  (Sinha  &  de  Week,  2012).  The  metric  is 
evaluated  using 

n  /  n  n 

C(n,  m,  A)  =  ^  at  +  I  ^  ^  PyAy 
«= i  \t=i 7=i 

where  n  is  the  number  of  components  in  the  system,  m  the  number  of  interfaces,  A  the 
DSM,  ai  the  complexity  of  each  component,  /?^-  =  fijCtiCCj  the  complexity  of  each  interface, 
Y  =  1/n  a  normalization  factor,  and  £04)  the  matrix  energy  of  the  DSM.  Although  the 
proposed  metric  is  very  sophisticated,  its  application  sees  the  evaluation  of  at  through 
expert  judgment,  and  ftj  =  1  for  lack  of  more  information  (Sinha  &  de  Week,  2013).  One 
interesting  feature  of  this  metric  is  the  topological  complexity  £04),  which  represents  the 
level  of  robustness  and  reliability  of  the  graph  network  and  can  be  easily  evaluated  from  the 
DSM  through  singular  value  decomposition. 

Hybrid  Structural-Behavioral  Complexity  Framework 

The  goal  of  this  research  is  to  develop  a  framework  for  the  identification  of 
complexity  level  of  the  engineered  system  and  architectural  patterns  affecting  the  behavior 
of  the  system  and  various  levels  of  risks.  The  framework  will  be  applied  at  the  initial  design 
phase,  when  system  requirements  are  defined,  and  the  system  architecture  is  in  its  initial 
development  (some  hierarchical  levels  are  defined  but  not  all  of  them). 

Our  suggested  framework  is  based  on  two  main  ideas.  The  first  one  is 
decomposition.  According  to  McCabe,  the  complexity  of  a  collection  of  unconnected  control 
graphs  is  equal  to  the  summation  of  their  complexities  (McCabe,  1976).  Wade  pointed  out 
that  in  complex  systems,  reduction  by  decomposition  cannot  work  since  the  behavior  of 
each  component  depends  on  the  behaviors  of  the  others  (Wade  &  Heydari,  2014).  This  is 
true  for  complex  engineered  systems,  but  in  this  research  we  are  tackling  logical  depth,  and 
therefore  we  assume  that  the  reductionist  approach,  as  described  by  Kauffman  (2007)  can 
be  applied  to  the  problem. 

The  second  idea  is  that  it  is  possible  to  measure  the  complexity  of  an  entity  at  its 
boundary.  We  have  seen  that  various  behavioral  complexity  metrics  have  been  proposed. 
These  metrics  consider  the  system  as  a  black  box  and  only  take  into  account  its  output.  In 
this  research  we  are  going  to  consider  not  the  output,  but  the  relationship  between  output 
and  input,  as  we  believe  it  better  describes  what  the  system  does. 

Framework  Application  Approach 

In  order  to  measure  the  system  complexity,  the  framework  will  combine  the 
complexity  of  components  that  make  up  the  subsystems  at  various  architectural  levels.  This 
combination  can  be  performed  applying  a  structural  complexity  metric,  which  considers  the 
system  architecture  (usually  represented  as  a  DSM  or  adjacency  matrix)  and  the  complexity 
of  each  component  at  a  certain  hierarchical  level.  The  complexity  of  a  subsystem  can  be 
evaluated  with  this  approach,  assuming  that  the  complexity  of  its  components  and  its 
internal  structure  are  known.  The  process  can  be  repeated  upwards  in  the  hierarchy  to 
evaluate  the  complexity  of  the  system. 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-238- 


At  this  point,  this  framework  can  use  all  the  other  structural  complexity  metrics 
already  available  in  literature.  The  existing  complexity  measures  in  literature  assume  that 
the  complexity  of  each  component  is  already  known,  or  if  that’s  not  the  case,  that  it  can  be 
evaluated  using  expert  judgment  or  historical  data.  In  the  creation  of  this  framework  we  have 
attempted  to  remove  the  majority  of  the  sources  of  subjectivity. 

Given  that  the  architecture  is  not  completely  defined,  there  will  be  some  components 
that  are  not  more  than  black  boxes.  The  complexity  of  these  components  can  be  measured 
with  behavioral  metrics.  Of  course,  historical  data  about  input  and  output  of  these 
components  in  past  projects  will  be  necessary  in  order  to  evaluate  the  metrics,  but  the 
subjectivity  coming  from  expert  judgment  will  be  removed.  Also,  there  is  a  difference 
between  using  historical  data  such  as  input  and  output,  which  for  engineered  systems  are 
physical  quantities,  and  historical  data  such  as  rate  of  failure,  or  schedule  delays  due  to 
integration,  which  depend  on  the  history  of  the  systems  they  are  derived  from. 

The  application  of  this  framework  can  be  divided  into  five  main  phases: 

1 .  The  architecture  needs  to  be  defined.  It  is  important  that  there  is  no 
connection  between  components  (or  functions)  at  different  levels,  or  even 
between  components  that  are  children  of  different  subsystems.  The  only  type 
of  connection  allowed  for  the  decomposition  principle  to  be  valid  is  between 
components  within  the  same  subsystem. 

2.  Once  the  architecture  is  defined,  it  is  necessary  to  characterize  the  boundary 
of  each  component.  The  interfaces  with  other  components  within  the  same 
subsystem  need  to  be  quantitatively  classified,  in  order  to  be  used  in  a 
behavioral  evaluation. 

3.  Once  the  interfaces  are  defined  and  characterized  according  to  their 
behavior,  the  complexity  of  each  black-box  component  can  be  evaluated 
using  a  behavioral  complexity  metric. 

4.  The  complexity  of  each  subsystem  is  then  evaluated  using  a  structural 
complexity  metric,  from  the  complexity  of  its  components  and  information 
about  its  internal  structure. 

5.  Once  the  complexity  of  the  lowest  level  components  (i.e.,  the  leaves  of  the 
hierarchy  tree)  is  evaluated,  it  can  be  combined  in  a  bottom-up  approach  to 
evaluate  the  complexity  of  the  higher  level  subsystems  by  repeating  the 
previous  steps  until  the  complexity  of  the  overall  system  is  evaluated. 

This  framework  has  been  built  with  flexibility  in  mind,  meaning  that  the  interface 
characterization  model,  the  behavioral  metric,  and  the  structural  metric  are  supposed  to  be 
plugged  in  according  to  the  specific  characteristics  of  the  enterprise  building  the  system,  and 
the  type  of  system.  We  have  attempted  to  remove  the  majority  of  the  subjectivity  from  the 
evaluation,  since  the  level  of  accuracy  depends  heavily  on  the  level  of  experience  of  the 
experts,  but  we  want  to  retain  the  knowledge  that  any  system  architect  has  about  the 
system  that  its  enterprise  is  comfortable  building.  Two  senior  system  architects  are  going  to 
evaluate  architectures  differently,  according  to  their  experience  and  the  experience  of  the 
people  they  worked  with,  thus  naturally  picking  the  best  choice  for  the  enterprise  they  work 
for.  Just  as  likely,  the  framework  can  be  adapted  to  rate  as  “better  designed”  the 
architectures  having  traits  that  the  enterprise  successfully  implemented  in  past  projects. 
Figure  3  shows  a  summary  of  the  hybrid  structural-behavioral  framework. 
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Hybrid  Structural- 
behavioral  complexity 
assessment  framework 


1.  Define  the  architecture  of  the 
engineering  system 

2.  Characterize  the  boundaries  and 
interfaces ofeach  component 

■ 

3.  Use  behavioral  complexity  metrics  to 
assess  the  complexity  of  each 
component 

4.  Evaluate  complexity  of  each 
subsystem 

% 

5.  Once  the  complexity  «f  the  lowest  level 
components  devaluated,  combine  in  a  bottom-up 
a  pproach  to  evalu  ate  the  complexity  of  the  h  jg  her 
level  subsystems,  by  repeating  the  previous  steps, 
un  J  the  complexity  of  the  overall  system  is 
evaluated. 


Figure  3.  Schematics  of  the  Hybrid  Structural-Behavioral  Complexity  Assessment 

Framework 

Part  of  this  research  effort  is  devoted  to  generating  the  modules  (interface  models, 
behavioral  metrics,  and  structural  metrics)  that  will  then  be  used  in  the  framework,  and  also 
to  understanding  which  set  of  modules  will  give  the  best  fit  for  each  specific  enterprise. 

Interface  Characterization  Model 

The  connections  between  the  components  of  an  engineered  systems  are  of  various 
natures  and  often  incommensurable.  For  example,  considering  two  components  having  a 
mechanical  and  a  thermal  interface:  Is  it  better  to  have  low  mechanical  stresses  and  high 
thermal  fluxes,  or  vice  versa?  In  order  to  answer  this  question,  the  interfaces  need  to  be 
classified  in  a  scale  that  allows  comparison  between  them  even  when  they  are  of  different 
natures.  This  will  enable  the  evaluation  of  many  structural  and  behavioral  metrics  that 
include  interface  complexity. 

Currently  this  model  is  still  under  refinement.  The  assumptions  are  based  on  the  idea 
that  connections  can  be  ranked  in  terms  of  how  enabling  they  are  towards  a  specific  goal. 

As  an  example,  consider  the  two  groups  of  animals  depicted  in  Figure  4. 
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Figure  4.  Herd  of  Sheep  and  Army  of  Ants 

Note.  These  two  groups  of  animals  are  examples  of  constraining  and  enabling  interactions. 

Both  the  herd  of  sheep  and  the  army  of  ants  are  a  group  of  animals  that  interact  with 
each  other.  Here  the  interaction  of  interest  is  the  purely  mechanical  one.  This  type  of 
interaction  is  constraining  in  the  case  of  the  herd,  since  it  decreases  the  degrees  of  freedom 
of  the  system.  This  also  happens  in  the  case  of  the  army  of  ants,  but  in  this  case  the  system 
has  gained  in  capabilities  (i.e.,  the  ability  to  bridge  in  mid-air).  The  emergence  of  this 
capability  is  given  by  the  enabling  nature  of  the  mechanical  connection.  The  goal  of  this  part 
of  the  research  regarding  interface  modeling  is  to  develop  a  metric  for  the  evaluation  of  the 
level  of  enablement  of  any  interface  towards  a  specific  component,  within  engineered 
systems. 

Use  Case:  Satellite  Attitude  Control  System 

In  order  to  show  how  the  framework  can  measure  the  complexity  of  a  system,  we 
have  applied  the  initial  framework  to  the  architecture  of  an  Attitude  Control  System  (ACS)  for 
a  satellite.  The  preliminary  architecture  is  represented  in  Figure  5. 


Figure  5.  Hierarchical  Representation  of  the  Architecture  of  the  ACS 

The  component  C.O,  in  this  case  the  ACS,  is  made  up  of  three  components — C.1 , 
C.2,  and  C.3 — which  are  the  attitude  sensors,  attitude  computer,  and  attitude  actuators, 
respectively.  For  the  sake  of  this  example,  the  architecture  of  the  component  C.2  has  been 
laid  out  only  for  its  software.  This  architectural  level  includes  components  C.2.1,  C.2. 2,  and 
C.2. 3,  namely  data  management  software,  quaternion  manipulation  software,  and 
proportional  control  software.  The  physical  architecture  presented  in  Figure  6  has  a  one-to- 
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one  mapping  with  the  functional  architecture,  and  therefore,  for  the  purposes  of  this 
example,  they  are  considered  as  equivalent. 


Figure  6.  IDEFO  Representation  of  the  F.O  Function  Corresponding  to  the  C.O 

Component,  the  ACS 

A  hierarchical  representation  of  the  system  architecture  is  not  enough  for  the 
application  of  the  framework.  The  interfaces  between  the  components  also  need  to  be 
defined.  Figure  6  shows  these  interfaces  within  the  F.O  function.  The  interactions  have  been 
defined  on  the  basis  of  four  use  cases:  attitude  maneuver,  safe  mode  attitude  maneuver, 
provide  attitude  parameters,  and  ACS  software  update.  The  information  reported  in  Figure  6 
allows  us  to  build  an  adjacency  matrix  for  the  components  of  C.O  that  can  be  used  in  the 
evaluation  of  any  structural  complexity  metric. 


0 

1 

O' 

^C.O  — 

1 

0 

1 

.0 

0 

0. 

In  this  example,  the  complexity  metric  proposed  by  Sinha  &  de  Week  (2013), 

n  /  n  n  \ 

C(n,m,A )  =  XHXX  PiAi  \yE(A)  (9) 

i=i  \i=iy=i  / 

will  be  used  to  evaluate  the  complexity  of  the  C.O  component  Cc  o.  In  this  case  at  =  Cc  i,  y  = 
1/3  can  be  evaluated  using  singular  value  decomposition  and  taking  the  sum  of  the  diagonal 
values  E(AC  0 )  =  1  +  V2.  Equation  9  then  becomes 

Q.o  =  Q.i  +  Q.2  +  Q.3 1  ^  (/?i2  +  @21  +  fi 23 )■  (10) 

Equation  10  still  has  many  unknown  variables,  which  need  to  be  computed.  /?12,  fi2i> 
and  /? 23  can  be  evaluated  using  the  interface  characterization  model.  The  evaluation  of  Cc2 
has  the  same  structural  approach  of  Cco,  since  its  internal  architecture  has  been  already 
defined.  The  hybrid  nature  of  this  framework  allows  consideration  of  the  most  information 
available,  evaluating  the  complexity  of  components  with  already  defined  internal  structure 
using  structural  complexity  metrics  that  take  the  aforementioned  structured  into  account. 

Cc  l  and  Cc  3  can  be  evaluated  using  a  behavioral  complexity  metric.  This  approach 
is  necessary  since  these  components  are  only  defined  as  black  boxes  and  we  only  have 
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information  about  their  input  and  output.  Evaluating  the  complexity  of  C.1  using  an  approach 
based  on  Chaisson’s  metric  is  taken  at  this  stage.  The  metric  considers  the  energy  that  the 
component  exchanges.  In  the  case  of  engineered  systems,  energy  can  be  exchanged  in  a 
variety  of  ways  (e.g.,  chemical,  data,  mechanical,  thermal).  The  evaluation  of  this  exchange 
is  also  part  of  the  interface  characterization  model  under  development  in  this  research. 

In  order  to  understand  the  dependency  of  the  structural  complexity  on  the  interfaces, 
we  can  modify  the  architecture  of  F.O  by  adding  a  connection  between  F.1  and  F.3.  In  this 
case,  the  new  component  C.O  will  have  an  adjacency  matrix: 


0 

1 

r 

Ac. O'  — 

1 

0 

1 

O 

0 

0- 

This  leads  to  a  different  value  of  the  matrix  energy  E(AC0> )  =  1  +  V3  and  thus  to  a 
new  formulation  for  the  complexity  of  the  component: 

Q.O  =  Q.l  +  Q.2  +  Q.3  "I  g  (/?12  +  Pl3  +  021  +  P23)  (12) 

This  change  in  the  architecture  increases  the  complexity  of  the  component.  Other 
structural  complexity  metrics  such  as  the  metric  proposed  by  Sinha  cannot  capture  this 
change  properly,  since  an  addition  of  a  single  connection  between  two  components  leads  in 
this  case  to  two  changes  in  the  complexity  evaluation.  For  this  reason,  in  this  research  we 
will  continue  to  propose  modifications  to  existing  complexity  metrics  so  that  the  overall 
framework  can  lead  to  more  meaningful  evaluations. 

Summary  and  Future  Work 

In  this  research  we  propose  a  framework  to  perform  a  quantitative  and  more 
objective  assessment  of  complexity  level,  as  a  major  precursor  to  assessing  objective 
technical  risks  and  failures  in  engineered  systems.  This  is  part  of  a  larger  research  vision 
and  objective  of  a  theoretical  model  of  failure  mechanisms  and  risks  in  engineered  systems, 
which  is  based  on  the  complexity  content  of  the  system.  This  part  of  our  research  focuses 
on  the  preliminary  design  phase  complexity  assessment  and  follows  and  builds  upon  the 
previous  work  by  Salado  and  Nilchiani  (2012)  on  the  complexity  assessment  of 
requirements  and  its  translation  in  risks  and  vulnerability  assessment.  The  new  framework 
suggested,  once  completed,  will  be  applicable  to  both  development  and  acquisition 
programs,  as  long  as  the  system  architecture  is  partially  available. 
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Likelihood 


Some  Problems  with  the  Current  Guidance 


"Risk  is  a  measure  of  future  uncertainties  in  achieving 
program  performance  goals  and  objectives  within  defined 
cost,  schedule  and  performance  constraints." 

-  Office  of  the  Undersecretary  of  Defense 


Consequence 

o  The  current  risk  identification  method  does  not  inform  the  decision  makers  well  on  the  underlying  causes  of  risk 
and  consequences. 


o  No  variation  (error  bars)  around  three  colors.  Abrupt  shift  from  one  color  to  other  is  possible  and  is  seen  in  practice. 
Interactions  and  ordering  among  risks  cannot  be  shown.  Consequences  are  not  presented  in  tangible  forms  of 
potential  cost  and  schedule  overruns  as  well  as  underperformance 

o  No  typology  of  risks  associated  with  causes  (internal,  external),  phases  of  life  cycle  (certain  risks  are  more  common 
in  particular  phases),  and  interconnections  among  choices. 


o  Consequences  are  not  presented  in  tangible  forms  of  potential  cost  to  remedy  (a  NASA  practice)  and  extent  of 
schedule  overruns.  PMs  cannot  use  risk  matrix  to  make  trades. 
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Different  Approaches 


Two  major  different  Approaches: 

1.  Incrementally  improve  the  existing  probability  based  assessment  methods  &  tools,  including  adaptation 
of  risk  assessment  methods  from  other  disciplines. 

2.  Investigate  and  examine  program  artifacts  for  roots  of  technical  risk.  These  in  many  instances  originate 
from  the  structure  and  architecture  of  the  system  or  from  the  organization  creating  the  system. 
Feedback  loops  and  existence  of  delays  are  a  few  of  the  examples  of  issues  that  are  often  the  deep 
sources  of  technical  risks.  Create  quantitative  measures  of  the  structure  of  the  system  and  correlate 
them  to  current  risk  measures  of  the  acquisition  program. 


Problem  Statement 


Emergence  of 
Failures  and  events 


Domain  of  Risk  identification  and  analysis: 

A  large  portion  of  risks  and  consequences 
internal  to  the  system,  are  observable  as 
symptoms  of  deeper  underlying  structure  of 
the  system 


Domain  of  Hidden  Structural  Complexity 
and  Dynamics,  vulnerability  and  fragility: 

Certain  signatures  and  behavior  rooted  in 
structure  of  the  technical  system  and/ or  the 
organization  cause  the  increased  risk  at  the 
surface  level. 
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Research  Approach 


Complex  Systems  Engineering  Dilemma 

Complexity  is  fragility  and  risk 

more  complex  — >  higher  likelihood  of  failure 

— >  more  difficult  to  manage 
— >  more  expensive  to  maintain 
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Complexity  is  value 


more  complex  — >  more  functions 

nctions 

unique  (emergent) 


Functional  Complexity 


2.5* 


Integral  System 
Modular  System 


Complex  systems  exhibit: 

■  Potential  for  unexpected  behavior 

■  Non-linear  interactions 

■  circular  causality  and  feedback  loops 

■  May  harbor  logical  paradoxes  and  strange 
loops 

■  Small  changes  in  a  part  of  a  complex  system 
may  lead  to  emergence  and  unpredictable 
behavior  in  the  system  (Erdi,  2008) 

■  Different  from  complicated  systems 

The  increased  complexity  is  often  associated 
with  increased  fragility  and  vulnerability  of  the 
system. 

By  harboring  an  increased  potential  for 
unknown  unknowns  and  emergent  behavior,  the 
probability  of  known  interactions  that  lead  to 
performance  and  behavior  in  a  complex  system 
decreases,  which  in  turn  leads  to  a  more  fragile 
and  vulnerable  system. 
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Research  Approach 


Concept  Program  Technology  Production  and 

Exploration  Definition  Development  Fielding 


Figure  11.  Complexity  evolution  throughout  the  systems  acquisition  lifecycle 
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Problem  Complexity  and  Requirements 


Problem 

Functional 

complexity 

complexity 

verall 

^^^^^omplexity> 

Organiz. 

Structural 

complexity 

complexity 

n 

H  =  -K-'YJPi‘ log  j(Pi) 

i= 1 


complexity  index 
functional  complexity  index 
organizational  complexity  index 
problem  complexity  index 
structural  complexity  index 


Functional  requirements  (Do) 

What  the  system  does  in  essence,  which  includes  what  it 
accepts  and  what  it  delivers 
Performance  requirements  (Being): 

How  well  the  system  does  it,  which  includes  performance 
related  to  functions  the  system  performs  or  characteristics  of 
the  system  on  its  own,  such  as  — ilities 
Resource  requirements  (Have): 

What  the  system  uses  to  transform  what  it  accepts  in  what  it 
[delivers 

Interaction  requirements  (Interact): 

Where  the  system  does  it,  which  includes  any  type  of 
operation  during  its  life-cycle.f 


C (Cp>  Cf>  cs,  C0 )  P(cpf  cy,  cs,  c0)  ■  log j  [P (cp>  c^, 


Cp  Cf  Cs  Co 


©2013  Salado  and  Nilchiani 


A  conflict  may  exist  when... 

...two  or  more  requirements  compete  for  the  same  resource. 

...two  or  more  requirements  oblige  the  system  to  operate  in  two  or  more 
phases  of  matter. 

...two  or  more  requirements  inject  opposing  directions  in  laws  of  society. 

...two  or  more  requirements  inject  opposing  directions  in  laws  of  physics. 


where  K  is  a  calibration  factor  that  allows  problem  complexity  to  be  adjusted  to  accurately 
reflect  an  organization's  business  performance.  The  first  term  represents  the  size  of  the 
requirement  set,  i.e.,  how  many  functional  requirements  r/the  system  has  to  fulfill.  These 
are  weighted  (a)  to  reflect  inherent  difficulty  of  requirements  and  adjusted  for  diseconomies 
of  scale  (£).  The  last  term  represents  complexity  modifiers  derived  from  amount  and  types  of 
conflicts  (/-/).  They  are  adjusted  to  reflect  influence  and  diseconomies  of  scale  ( b ). 


©2013  Salado  and  Nilchiani 


The  spacecraft  was  a  partially  reusable  human  spaceflight  vehicle  for  Low  Earth 
Orbit,  which  resulted  from  joint  NASA  and  US  Air  Force  efforts  after  Apollo.  "The 
vehicle  consisted  of  a  spaceplane  for  orbit  and  re-entry,  fueled  by  an  expendable 
liquid  hydrogen/liquid  oxygen  tank,  with  reusable  strap-on  solid  booster  rockets. 

[...]  A  total  of  five  operational  orbiters  were  built,  and  of  these,  two  were  destroyed 
in  accidents." 


"Soyuz  is  a  series  of  spacecraft  initially  designed  for  the  Soviet  space  programme 
and  still  in  service  today.  [...]  The  Soyuz  was  originally  built  as  part  of  the  Soviet 
Manned  Lunar  programme.  [...]  The  Soyuz  spacecraft  is  launched  by  the  Soyuz 
rocket,  the  most  frequently  used  and  most  reliable  Russian  launch  vehicle  to 
date." 
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Problem  Complexity: 
Shuttle  vs.  Soyuz 
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Alternatives 


Hybrid  Structural- Behavioral  ^ 

Complexity  Framework 


r 


Behavioral  Complexity 
Metrics 

•  Based  on  the  behavior  of  the  system 

•  Evaluate  the  complexity  of  the 
output 

•  Many  examples  in  existing  literature 


Structural  Complexity 
Metrics 

•  DSM  Based 

•  Evaluate  the  complexity  of  the 
architecture 

•  Many  examples  in  existing  literature 


Interface  Characterization 
Model 


•  Way  of  comparing  incommensurable 
interfaces 

•  Looks  at  the  effect  of  the  interface 

•  Ranks  interfaces  based  on  the  level 
of  enablement 
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Hybrid  Structural- Behavioral  ^ 

Complexity  Framework 


•  Define  the  architecture  of  the  engineered  system 


•  Characterize  the  boundaries  and  interfaces  of  each  component 


•  Use  behavioral  complexity  metrics  to  assess  the  complexity  of  each 
component 

•  Use  structural  complexity  metrics  to  evaluate  the  complexity  of  each 
subsystem 

•  Repeat  the  previous  steps  to  evaluate  the  complexity  of  higher  level 
subsystems 
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Structural  Complexity  Metrics 
McCabe  (1976) 


Complexity  metric  v(G): 

v(G )  =  e  —  n  +  2  p 

•  e  is  the  number  of  edges 

•  n  is  the  number  of  vertices 

•  p  is  the  number  of 
connected  components 


M: 


v(MAB )  =  13-13  +  2  *3  =  6 
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Structural  Complexity  Metrics  ^ 

Cotsaftis  (2009) 


Complexity  metric  Cs : 

Cs  =  n/N 

•  N  is  the  total  number  of  nodes  in  the  system 


Exterior 

Component 


n  is  the  number  of  components  that  satisfy  the 
inequality 

inf  Pi  j  »  Pi  i,  Pie  : 

Pij  is  the  flux  of  resource  from  node  i  to  node  j  ; 


Pa  is  the  generation  or  usage  of  resource 
for  node  i 

Pie  is  the  resource  flux  from  node  i  to  the 
environment 


Node  j 


System  Boundary 

Fig.  2  :  Graph  Representation  of  System  with 
Its  Three  Exclusive  Types  of  Vertices  Vfl,  Vu  and  Vy 
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Structural  Complexity  Metrics 
Sinha  &  deWeck  (2012) 


Complexity  metric  C(n,m,  A) : 

n  inn  \ 

C(n,m,A )  =  +  I  'Yj'Yj/3ijAij  \yE(A) 

i=i  \i=i  j=i  ) 

•  n  is  the  number  of  components 

•  at  is  the  complexity  of  each  component  i 

•  fiij  is  the  complexity  of  the  interface  between  components  i  and  j 

•  A  is  the  adjacency  matrix  of  the  system 

•  y  =  1/n 

•  E(A)  is  the  energy  of  the  adjacency  matrix  which  is  the  sum  of  the  singular 
values  of  A,  evaluated  through  singular  value  decomposition 
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Interface  Characterization  Model  ^ 
Enablement  and  Constraint 

Components  in  engineered  systems  are 
connected  to  other  components  so  they  can 
either  do  thinghs  they  can’t  do  alone 
(enablement),  or  so  that  they  cannot  do  things 
they  would  otherwise  do  (constraint). 

Assumption:  for  each  interface  between  two 
components  the  level  of  enablement/constraint 
that  a  component  exercises  on  the  other  can  be 
measured. 

The  model  will  quantitatively  rank  interfaces 
based  on  the  level  of  enablement/constraint, 
independently  from  their  nature  (e.g. 
mechanical,  thermal,  chemical, 
electromagnetic). 

http://thatscienceguv.tumblr.conn/post/48996081962 
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Behavioral  Complexity  Metrics  ^ 

Chaisson  (2004) 


Chaisson  just  provides  a  definition  for 
this  metric  as,  free  energy  rate 
density,  which  is  energy  entering  the 
system  per  unit  of  time  per  unit  of 
mass. 

He  did  although  evaluate  its  value 
for  many  entities  in  the  universe. 

The  accurate  trend  leads  to  think 
that  a  metric  based  on  this  concept 
could  be  useful  in  the  measurement 
of  complexity  for  engineered 
systems. 
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Behavioral  Complexity  Metrics 
Willcox(2011) 

Complexity  metric  C(Q): 

C(Q)  =  exp  (h(X)) 


h(X)  =  -  fx(x)  log  fx(x)  dx 
Jax 

•  X  is  the  joint  distribution  of  the  quantities  of  interest 

•  /i(X)  is  the  differential  entropy  of  X 

•  £lx  is  the  support  of  X 

•  fjs  the  pdf  of  a  specific  distribution 


This  metric  shows  how  the  framework  would  be  able  do  accommodate 
uncertainty  at  the  component  level. 
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Use  Case:  ^ 

Satellite  Attitude  Control  System 


We  are  going  to  show  the 
application  of  the  framework  using 
the  structural  complexity  metric 
proposed  by  Sinha  &  deWeck 
(2012). 

The  evaluation  of  the  complexity  of 
the  component  C.O  is  performed 
using  the  components  at  the  1st  level 
C.l,  C.2,  and  C.3. 


n 


C(n,  m,  A)  = 


i-iii* 


lAj 


i= 1 


i= 1  y=l 


JyE(A) 


'c ~ 

'C3- 

Attitude  Sensors 

Attitude 

Computer 

Attitude 

Actuators 

Component 

Component 

Component  ^ 

C.2.1 

C.2.2 - 

C.2. 3 

Data 

Management  So... 

Quaternion 
Manipulation  So... 

Proportional 
Control  SoftiA'are 

Component 

Component 

Component 
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Use  Case:  # 

Satellite  Attitude  Control  System 


Reaction  Torque 
ACS  Software  Update 
Request  for  Attitude  Parameters 


Request  fbr  Attitude  Maneuver 
Safe  Mode  Alert 


Energy 


ACS 


0 

1 

a 

II 

0 

Cj 

1 

0 

1 

.0 

0 

a 

1  +  V2 

Cc. 0  =  Cc.  1  +  Cc. 2  +  Cc. 3  H - ^  (ft  2  +  fti  +  P23) 
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Use  Case: 

Satellite  Attitude  Control  System 


Cc. 0  —  Cc.l  +  Cc. 2  +  O'. 3  "I - g  (/?12  +  p2l  +  ^23) 


1  +  V2 


The  missing  terms  in  the  equation  above  cannot  be  evaluated  in  the  current 
state  of  the  framework. 

The  complexity  of  the  components  is  going  to  be  evaluated  using 
behavioral  metrics,  using  historical  information  about  input/output  of  the 
components.  In  our  opinion  this  is  better  than  using  historical  complexity/ 
reliability/robustness  data,  since  do  not  depend  on  the  history  of  the 
specific  components. 

The  complexity  of  the  interface  is  going  to  be  evaluated  using  the  interface 
characterization  model. 
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Modification  of  Existing  Metrics 
Sinha  &  deWeck  (2012) 


Reaction  Torque 
ACS  Software  Update 
Request  for  Attitude  Parameters 


Request  fbr  Attitude  Maneuver 
Safe  Mode  Alert 
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Modification  of  Existing  Metrics  ^ 

Sinha  &  deWeck  (2012) 


n  /  n  n  \ 

C(n, m,A)  =  +  |  '^^ptjAiJ  JyF04) 


Q.o  —  Qt.1  +  Q.  2  +  Q;.3  + 


i=l 7=1 

1  +  V2 


(/^12  +  /?21  +  P23) 


CC.0r  -  Q.l  +  Q;.2  +  Q.3  "t - +(^L3)+  /?21  +  P23) 

Following  the  addition  of  one  connection  between  C.l  and  C.3  the  metric 
has  a  twofold  change.  We  propose  the  following  modification  to  this  metric: 

n 


C  ( n ,  m,  A)  =  ^  cci  +  yE  ( B ) 

i=l 


where  B  is  the  matrix  whose  elements  are  ptj. 
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Summary  and  Future  Work 


In  this  work  we  introduced  the  Hybrid  Structural-Behavioral  Complexity 
Framework. 

The  framework  backbone  has  been  defined,  but  its  modules  are  yet  to  be 
developed. 

Some  modules  are  to  be  developed  by  modifying  existing  complexity 
metrics,  while  others  are  to  be  developed  ex  novo. 

Future  work  will  focus  on  the  development  of  those  modules  and  the 
validation  of  the  framework  using  real  data. 
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Thank  you  for  your  attention 


Questions? 


Backup  Slides 

Example:  DARPA  F6  Program 


Context:  The  Need  for  Adaptability  and 
Resilience  in  Space  Systems  In  Uncertain  World 


Space  Systems: 

-  Lengthy  design  and  manufacturing 

-  Long  lifetimes 

-  Very  expensive 

-  Limited  access  after  launch 

-  Face  extensive  uncertainties  during 
their  lifetime 


Space  systems  often  provide  a  good 
response  to  initial  requirements  but: 

-  They  fail  to  meet  new  market  conditions 

-  They  cannot  adapt  to  new  applications 

-  Their  technology  becomes  obsolete 

-  They  cannot  cope  with  changes  in  context/ 
environment  (markets,  policy,  technological 
innovation,  changing  human  needs) 


Large  upfront 
costs  and  hard- 
budget 
environment 


Lack  of  coherent 
way  to  measure 
value  of 
Adaptability 


Uncertain/ 
ambiguous 
return  on 
investment 


— V 


Lack  of 
implementation 
and  Design  of 
flexibility  in 
Space  Systems 


p 


Response  to  change: 

Through  change  of 
software,  the  low  gain 
antenna  was  used  to 
transfer  the 
information  back  to 
Earth.  Instead  of  a 
total  mission  failure, 
70%  of  the  original 
mission  goal  was 
achieved. 


Response  to 
change: 

None. 

Iridium  failed  to 
respond  to 
changes  in  the 
market  and  filed 
for  bankruptcy. 


A  low  gain  antenna  was 
designed  into  the  system. 


Change: 

A  large  market 
decrease  from  a 
predicted 
400,000  to 
50,000 
subscribers 


A  large  and  fixed  capacity 


Change: 

Galileo’s  high 
gain  antenna 
failed  to  open. 
The  information 
could  not  be 
transferred  back 
to  Earth. 
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Infrastructure/Bus  Support 

Low  c  *•  rv  *  u  *•  High 

Function  Distribution 


An  Overview  of  a  Fractionated  Spacecraft  Concept 


Status  quo 


Payload  separation  with  no  resource 
sharing  or  closed-loop  cluster  flight 


«► 


Enablers  of  Fractionated  Space 
Architectures 

•  Cluster  maintenance 

•  Rapid  cluster  maneuvering 

•  Relative  navigation 

•  Wireless  networking 

•  Real-time  resource  sharing 

•  Multi-level  security 

•  24/7  LEO-ground  connectivity 

•  Open  F6  Developer's  Kit 

•  Modular  F6  Tech  Package 

•  Adaptability  Metrics 

•  Design-for-Adaptability  Tools 


Low 


Mission/Payload 
Function  Distribution 


High 


Credit:  Mr.  Eremenko,  DARPA 


Value  of  Adaptability  Under  Risk  and  Uncertainty 


What  is  the  quantitative  value  of  Adaptability  in  fractionated  spacecrafts? 

Integrating  various  systems  "ilitis"  into  a  single  framework  in  the  presence  of  multi¬ 
dimensional  uncertainty  using  scenarios  and  Real  options 

Space  System  Partial 
Failure/  Malfunction 


What  is  the  physical,  temporal,  and  logical 

Boundaries  of  the  Space  Systems  Under 
Study? 

What  are  the  types  of  Uncertainties  (risks  and 
opportunities)  a  Space  System  is  facing,  and 
how  they  manifest  themselves?  (Scenarios) 

What  are  Stakeholders  preferences  on 
Requirements  and  Utilities  of  the  space 
mission? 


What  are  the  Real  Options  in  and  on  space 
systems  and  how  to  model  them? 


Space  System 


Impact 


Technology 
Change 


Market  Change 


Environmental 
hange  and  Effects 


Other  Changes 
and  Factors 


2 


Economic 


Dimimwon 
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Uncertainty  Science,  Characterization  and  Modeling 


1.  Classifies  all  types  of  relevant  Space 
Systems  Uncertainties 

2.  Relevant  Models  for  each  type  of 
uncertainty 

3.  Uncertainty  is  plugged  into  real  options 
for  Adaptability  Measurement 


T 


Objective  vs. 

Endogenous 

Roots  and 

Subjective 

vs.  Exogenous 

Sources 

1 


Object  of 
Variation/System 
Aspects  of 
Uncertainty 
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Uncertainty  Science,  Characterization  and  Modeling 


F6  System  Boundaries 


Exogenous: 

•  Launch 

•  Orbital  Debris 

•  Space  hazards 

•  Market 

•  Funding 


I 

Endogenous: 

•  Module  Failure 

•  Component  failure 

•  Supply  Chain  delay 

•  Schedule  Uncertainty 

•  Change  in  user  needs  | 


I 


Modeling  and 
solution  to  address 
complex  Uncertain 


Agent  interference 


Complex  Uncertainty 


•  The  Adaptable  Response  creates  a  new 
uncertainty  profile  or  a  new  type  of 
|  uncertainty 


Complicated/Simple 

Uncertainties 


The  Adaptable  response  can  potentially 
respond  and  resolve  the  uncertainty  at 
hand 


Work 

Done 
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Uncertainties  and  Complexities  in  Space  Systems 


Modeling  Single  Complex  Uncertainty 


F6  Project  Requirement 
Uncertainties 


Funding 

Uncertainty 


Stakeholders 

Inputs 


Uncertainty 


Cost 


Requirement  Uncertainty  is  mainly  a  function  of  changing  user  and  stakeholders  need,  funding 
uncertainty,  and  incomplete  or  unclear  set  of  initial  requirements.  There  are  delays  in 
requirement  gathering  and  classification  and  prioritization  process  and  several  loops  of  iterations 
that  affect  cost  and  project  schedule  dramatically 
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Uncertainties  and  Complexities  in  Space  Systems 


On-going  Research:  Multiple  Uncertainties,  Realistic  Scenarios  and 
Catastrophic  failures 

-  Correlation  between  various  space  systems-related  uncertainties 

-  Realistic  Scenarios:  manifestation  of  a  uncertainty  and  chain  reaction  effect  of 
triggering  other  uncertainty  types,  Time  lag  between  Uncertainties  (Window  of 
opportunity  of  options) 

-  Correlation  of  increasing  in  complexity  measure  and  structural  complexity  of  the 
F6  and  catastrophic  chain  of  Uncertainties  (Murphy's  Law!) 


Escalation 


Group 


Uncertainties 


Policy 


Export,  frequency  allocation,  mission-specific 
regulations,  disposal. 


Technology 


Obsolescence,  technology  readiness,  system  readiness. 


Organization 


Supply  chain,  cost,  technical  capability,  key  people, 
V&V,  design,  requirements,  customer  involvement. 


Service 

performance 


Reliability,  availability,  space  debris,  space  radiation, 
weather  hazard,  lifetime,  performance. 


Market 


Market  size,  discount  rate,  competition,  market 
capture,  schedule. 
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Propagation  of  Failure  in  F6  Network  and 
correlation  with  Complexity  measure  of  the 
Network 


The  Less  Complexity  in  Design  Structure  and  Architecture  of 
F6,  The  slower  the  propagation  of  specific  types  of 
uncertainty  in  the  F6  architecture,  the  more  time  to 
interfere  and  respond  and/or  exercise  Real  Options, 
Therefore  More  Adaptability 


Correlation  Matrix 
of  Space  Systems 
Related 
Uncertainties 


Complexity  and  Uncertainty  in  F6:  Uncertainty 

Correlations 


Why  Uncertainty  Correlation  matters? 

-  Realistic  Scenarios,  Realistic  Options,  Time  to  Exercise  and  Option 

-  Trigger  possibility,  Chain  reaction  effect 
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Rare  catastrophic  events  in  complex  systems  are  poorly  probable  yet  highly 
possible!!  The  collective  effect  of  insignificant  uncertainties  have 
grave  consequences.  In  the  end  it  is  hard  to  figure  out  what  went  wrong! 
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Uncertainties  and  Complexities  in  Space  Systems 


r 

1 

■ 

Markel 

> 
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1 
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1 

Policy 

L 

1 

Category 

Description 

Policy 

Uncertainties  related  to  law  and  regulation  that  impact  the  system.  Most 
common  examples  include  ITAR,  EO  laws,  or  ITU  frequency  allocation. 

It  is  important  to  mention  that  uncertainties  falling  under  this  category  have 
not  really  been  explored  in  the  available  literature.  When  discussing  Policy 
uncertainty,  it  is  normally  related  to  government  funding,  which  we  allocate  to 
market. 

Technology 

Uncertainties  that  are  related  to  the  availability  of  technology  or  technical 
solutions.  Most  common  examples  are  obsolescence,  state-of-the-art, 
achievability,  TRL,  SRL,  etc. 

Organization 

Uncertainties  that  are  related  to  the  organization  of  the  system  (project)  and 
may  impact  the  development  or  the  operation  of  the  system.  Most  common 
examples  include  supply  chain,  complexity  of  operations,  directives  to  use 
specific  suppliers,  loss  of  key  personnel,  inadequate  personnel,  etc. 

It  is  important  to  mention  that  uncertainties  falling  under  this  category  have 
never  been  looked  into  in  the  available  literature. 

Service 

performance 

Uncertainties  that  are  related  to  the  impacts  of  bringing  the  system  into  real-life 
operation.  They  could  be  defined  also  as  uncertainties  included  in  the  design  by 
definition  (performance  based  on  probabilities).  Most  common  examples  may 
include  reliabilities,  availabilities,  TX  power,  degradation,  lifetime,  orbit 
accuracy,  fuel  usage,  radiation,  atmospheric  effects,  network  load,  integration  to 
other  systems,  etc.). 

Market 

Uncertainties  related  to  “funding  and  revenues”,  which  may  be  impacted  by 
business  case  success  or  effects  of  internal  and  external  competitors: 

Commercial  project:  market  capture,  effect  of  other  company  putting  the  system 
in  place  earlier  or  at  lower  cost,  impact  of  competitors  with  same  service  in 
other  industry  (e.g.  terrestrial  networks). 

Government  project:  actual  scientific  return,  competitors  making  funding 
fluctuate  (e.g.  budget  moved  from  Human  spaceflight  to  Earth  observation),  etc. 
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Structural  vs.  Functional  Complexity 


The  Simple 


The  Complex 


Single  cause  and  single  effect 

A  small  change  in  the  cause  implies  a  small  change  in  the  effect 
Predictability  and  Modelability 


Circular  causality,  feedback  loops,  logical 
paradoxes,  and  strange  loops 
Chaos:  small  change  in  the  cause  implies 
dramatic  effects 

Emergence,  unpredictability  and  entropy 


Complex  Systems  Engineering  Dilemma 

Complexity  is  fragility  and  risk 

more  complex  a  higher  likelihood  of  failure 
a  more  difficult  to  manage 
a  more  expensive  to  maintain 

Complexity  isvalue 

more  complex  a  more  functior 
abetter  functions 
aunique  (emergen#)  functions 
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Functional  Complexity 

Complexification  driving  force 


Emergence 


Exist  in  the  whole  not  in  the  parts 
Cannot  be  modelled 

In  complex  systems  failure  can  be  emergent 

Structural  Complexity  is  the  potential  for  and  intensity  of 

emergence 

It  is  important  to  measure  complexity 


Research  Approach 
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Complexity 


Uncertainty  and  Complexity  in  F6:  Catastrophic 

Failure 


Low  rate  of  complexity  increase  provides 
a  response  time  window 


Response  time 
window  ->  0 


Point  of  no  return 


Uncertainty  magnification: 

Collective  system  tolerance  of  most 
insignificant  uncertainties  ->0 


Collective  Uncertainty  Gi! 


Area  of  rapidly  increasing 
complexity 


Increased  structural  complexity  means 
quicker  failure  propagation,  shorter  time 
to  failure 
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Engineered  system  modeled  by  a  Discrete 
non-linear  Markov  process: 
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Failure  Propagation  Overview:  Time  To  Failure  Concept 


§Failure  propagation  as 
precursor  model 
§Affects  Complexity, 
vulnerability  and 
Adaptability  of  F6 
§Used  in  calculation 
options  values  in  face  of 
various  failures 
§Will  be  used  in  Security 
enhancement  options 
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Uncertainty  and  Complexity  in  F6:  Failure  Propagation 


Choose  1 

Choose 

Number  of 

Links 

subsystems 

density 

Rare  catastrophic  events  in  complex  systems  are  poorly  probable  yet  highly  possible!!  The 
collective  effect  of  insignificant  uncertainties  have  grave  consequences.  In  the  end  it  is  hard  to 
figure  out  what  went  wrong! 
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Uncertainty  and  Complexity  in  F6:  Failure  Propagation  vs 

Various  Number  of  Fractions 


Propagate  failure  inside  of  each 
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collective  effects  based  on  average 
rates 
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Failure  Propagation:  Results  and  Insights 


Insights: 

Our  goal  in  to  increase  TTF,  since  it  gives  us  more  time  to  detect  and  remedy 
failures  before  they  become  detrimental  to  the  whole  F6  architecture 
•Correlation  of  number  of  modules  and  Complexity  measure  of  the  system: 
Monoliths  often  have  the  least  structural  complexity 
•Mean  Time  to  Failure  decreases  with  number  of  fractions  and  modules  for 
majority  of  module  architectures 

•F6  architectures  with  higher  complexity  measures  are  more  vulnerable  and 
prone  to  catastrophic  failures 

•The  art  of  module  making:  maximum  cuts  creates  high  degree  of  coupling 
between  fractions  and  therefore  higher  complexity 


Sensitive  to  initial  partial  failure  locations,  modular  systems  can  be  extremely 
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Failure  Propagation:  Results  and  Insights 


Failure  detection  in  modular  structure  is  easier 


Insights: 

Failure  propagation  and  detection  in  various  F6  architecture  vs.  a  monolith 

•In  monoliths,  failure  propagates  at  a  very  slow  rate  initially  and  after  a  certain  level,  it  grows  exponentially 
•  In  modular  systems,  failure  propagates  rather  faster  initially,  but  grows  steadily 

•If  detectability  of  failure  is  defined  at  x%  (e.g,  10%),  Fractionated  systems  show  partial  failure  sooner,  as  well  as 
provide  decision-makers  with  time  to  react  (window  of  opportunity)  to  exercise  an  option  to  address  the  problem. 
In  many  monoliths,  when  the  failure  becomes  detectable  that  its  already  too  late 
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